Are Program Participants Good Evaluators

How well do program participants assess program performance ex-post? In this paper we compare participant evaluations based on survey responses to econometric impact estimates obtained using data from the experimental evaluation of the U.S. Job Training Partnership Act. We have two main findings: First, the participant evaluations are unrelated to the econometric impact estimates. Second, the participant evaluations do covary with impact proxies such as service intensity, outcome levels, and before-after outcome differences. Our results suggest that program participants behave as ‘lay scientists’ who seek to estimate the impact of the program but face cognitive challenges in doing so. JEL Codes: I28, J24, C83 Word count (main text): 10,927 (file) 14,216 We thank the W.E. Upjohn Institute for Employment Research for funding this research. Corresponding author: Jeffrey Smith, Department of Economics, University of Michigan, 238 Lorch Hall, 611 Tappan Street, Ann Arbor MI 48109 USA, 734-764-5359, [email protected]. We are very grateful to Iwan Barankay, Dan Black, Tanya Byker, Sebastian Calónico, Pam Giustinelli, Hilary Hoynes, Guy Michaels, Bob Schoeni, and audience members and discussants at Chicago (CHAS), Concordia, Indiana, LSE, Mannheim, Queen’s, UCL (especially Richard Blundell, Hidehiko Ichimura and Costas Meghir), Warwick, Notre Dame, the Quantitative Methodology Program (QMP) seminar at Michigan, the 2004 APPAM meetings in Atlanta, the 2004 SEA meetings in New Orleans, the 2006 SOLE meeting in Boston, the 2006 IZA/CEPR ESSLE in Amersee, and the 2006 IZA/IFAU Conference on Labor Market Policy Evaluation in Uppsala for helpful comments. We thank participants in a 2002 OECD conference on evaluating local economic development programs for inspiration. Any and all remaining “errors” are actually part of the subgame-perfect equilibrium strategy of a larger game.